<html>
<head><meta charset="utf-8"><title>How the macro parser implemented? · t-compiler/wg-rustc-dev-guide · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/196385-t-compiler/wg-rustc-dev-guide/index.html">t-compiler/wg-rustc-dev-guide</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How.20the.20macro.20parser.20implemented.3F.html">How the macro parser implemented?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="192117453"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How%20the%20macro%20parser%20implemented%3F/near/192117453" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lzx <a href="https://rust-lang.github.io/zulip_archive/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How.20the.20macro.20parser.20implemented.3F.html#192117453">(Mar 28 2020 at 10:52)</a>:</h4>
<p>The rustc dev guide just talk about its interface and it is really difficult for me to know its details.I want to know is:<br>
I 'm wondering how macro parer implemented. But i'm confused about in its' essential function parse_tt(parser, ms) in <a href="http://macro_paresr.rs" title="http://macro_paresr.rs">macro_paresr.rs</a>, What's parser is about?: what it's token stream is,? is it about invocation or defintion?  ,and What's ms (the matcher )contained?, if give me a example  below:</p>
<div class="codehilite"><pre><span></span>#[macro_export]
macro_rules! yo {
    ($name: expr) =&gt; {
        println!(&quot;Yo {}!&quot;, $name);
    }

}

fn main() {
    yo!(&quot;hello&quot;);
}
</pre></div>


<p>How can I figure out it?</p>



<a name="192117952"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How%20the%20macro%20parser%20implemented%3F/near/192117952" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How.20the.20macro.20parser.20implemented.3F.html#192117952">(Mar 28 2020 at 11:08)</a>:</h4>
<p>I think ms consists of the arms of the macro_rules eg <code>[($name:expr)]</code> in this case.</p>



<a name="192119263"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How%20the%20macro%20parser%20implemented%3F/near/192119263" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lzx <a href="https://rust-lang.github.io/zulip_archive/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How.20the.20macro.20parser.20implemented.3F.html#192119263">(Mar 28 2020 at 11:47)</a>:</h4>
<p>And parser contains yo!("hello"); in this example?@bjorn3</p>



<a name="192122886"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How%20the%20macro%20parser%20implemented%3F/near/192122886" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How.20the.20macro.20parser.20implemented.3F.html#192122886">(Mar 28 2020 at 13:22)</a>:</h4>
<p>I believe the parser contains just the part between braces: <code>"hello"</code></p>



<a name="192124656"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How%20the%20macro%20parser%20implemented%3F/near/192124656" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lzx <a href="https://rust-lang.github.io/zulip_archive/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How.20the.20macro.20parser.20implemented.3F.html#192124656">(Mar 28 2020 at 14:00)</a>:</h4>
<p>What api should i use to log it. I log about parser's tokenstream and ms's tokens but it is too hard too read.</p>



<a name="192438514"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How%20the%20macro%20parser%20implemented%3F/near/192438514" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How.20the.20macro.20parser.20implemented.3F.html#192438514">(Mar 31 2020 at 19:32)</a>:</h4>
<p>Yes, <span class="user-mention" data-user-id="133247">@bjorn3</span> is correct</p>



<a name="192438600"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How%20the%20macro%20parser%20implemented%3F/near/192438600" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How.20the.20macro.20parser.20implemented.3F.html#192438600">(Mar 31 2020 at 19:33)</a>:</h4>
<blockquote>
<p>What api should i use to log it. I log about parser's tokenstream and ms's tokens but it is too hard too read.</p>
</blockquote>
<p>Not sure what you mean?  Are you trying to log the contents of ms and tokenstream with <code>log::debug</code> or something? That's probably what I would do</p>



<a name="192613165"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How%20the%20macro%20parser%20implemented%3F/near/192613165" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lzx <a href="https://rust-lang.github.io/zulip_archive/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How.20the.20macro.20parser.20implemented.3F.html#192613165">(Apr 02 2020 at 01:55)</a>:</h4>
<p>Yes, this is what i want to do, I need something which can help me figure out how macro work flow,  which can log its tokenstream and data flow, maybe like a debugger,? i don't know  <span class="user-mention" data-user-id="198054">@mark-i-m</span></p>



<a name="192711608"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How%20the%20macro%20parser%20implemented%3F/near/192711608" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> mark-i-m <a href="https://rust-lang.github.io/zulip_archive/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How.20the.20macro.20parser.20implemented.3F.html#192711608">(Apr 02 2020 at 19:09)</a>:</h4>
<p>Hmmm... sorry one more clarification: are you trying to write a macro and want to debug why it's not doing what you expect? Or are you trying to modify the compiler. Unfortunately, debugging support for <code>macro_rules</code> is not great these days.</p>
<p>If you want to understand what the macro_rules parser does, I would recommend reading the comments in <a href="https://github.com/rust-lang/rust/blob/master/src/librustc_expand/mbe/macro_parser.rs" title="https://github.com/rust-lang/rust/blob/master/src/librustc_expand/mbe/macro_parser.rs">https://github.com/rust-lang/rust/blob/master/src/librustc_expand/mbe/macro_parser.rs</a> and you could insert <code>log::debug</code> calls in that file to get the contents of various matchers and token streams. e.g. You can print the matchers (<code>ms</code>) here: <a href="https://github.com/rust-lang/rust/blob/537ccdf3ac44c8c7a8d36cbdbe6fb224afabb7ae/src/librustc_expand/mbe/macro_parser.rs#L621" title="https://github.com/rust-lang/rust/blob/537ccdf3ac44c8c7a8d36cbdbe6fb224afabb7ae/src/librustc_expand/mbe/macro_parser.rs#L621">https://github.com/rust-lang/rust/blob/537ccdf3ac44c8c7a8d36cbdbe6fb224afabb7ae/src/librustc_expand/mbe/macro_parser.rs#L621</a> and you could print what is matched by printing these values: <a href="https://github.com/rust-lang/rust/blob/537ccdf3ac44c8c7a8d36cbdbe6fb224afabb7ae/src/librustc_expand/mbe/macro_parser.rs#L636-L640" title="https://github.com/rust-lang/rust/blob/537ccdf3ac44c8c7a8d36cbdbe6fb224afabb7ae/src/librustc_expand/mbe/macro_parser.rs#L636-L640">https://github.com/rust-lang/rust/blob/537ccdf3ac44c8c7a8d36cbdbe6fb224afabb7ae/src/librustc_expand/mbe/macro_parser.rs#L636-L640</a> (I think)... I'm a bit rusty on this code (no pun intended).</p>
<p>If you want to understand how procedural macros work, I'm afraid I can't really help, as I know almost nothing about how they work. Perhaps reading dtolnay's notes might help?: <a href="https://github.com/dtolnay/syn" title="https://github.com/dtolnay/syn">https://github.com/dtolnay/syn</a></p>



<a name="192748224"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How%20the%20macro%20parser%20implemented%3F/near/192748224" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> lzx <a href="https://rust-lang.github.io/zulip_archive/stream/196385-t-compiler/wg-rustc-dev-guide/topic/How.20the.20macro.20parser.20implemented.3F.html#192748224">(Apr 03 2020 at 01:11)</a>:</h4>
<p>thanks, It is really helpful for me . <span class="user-mention" data-user-id="198054">@mark-i-m</span></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>